Method and apparatus for extracting face feature

ABSTRACT

A method and apparatus for extracting a face feature to construct a face search system having a high recognition rate are provided. The method includes calculating an average image of each of a plurality of candidates having one or more face images to which different weights are given according to an order in which the face images are acquired and calculating a total average image of all face images of the plurality of candidates, calculating a between-class scatter matrix based on a difference between each candidate&#39;s average image and the total average image and a within-class scatter matrix based on a difference between each candidate&#39;s average image and each candidate&#39;s predetermined face image of the one or more face images, and generating a base vector that maximizes a criterion defined by a ratio between the between-class scatter matrix and the within-class scatter matrix.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application claims the benefit of Korean Patent Application No.10-2006-0122616, filed on Dec. 5, 2006, in the Korean IntellectualProperty Office, the disclosure of which is incorporated herein in itsentirety by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a method and apparatus for extracting aface feature, and more particularly, to a method and apparatus forextracting a face feature, wherein a weight is set according to theorder of acquiring images and a face feature is not affected by thenumber of image data of a candidate so that a face search system has ahigh recognition rate.

2. Description of the Related Art

A biometric system such as face recognition performs authentication andidentification according to a type of service. In authentication, theidentity of a user is confirmed using a one-to-one comparison method. Inidentification, the identity of a most similar person to a predeterminedperson is confirmed using a one-to-many biometric data comparison for aplurality of persons registered in a database. That is, inauthentication, a binary class (or two class) result value is generatedas yes or no, whereas in identification, a list of candidates isgenerated in the order of the decreasing probability as a result value.

Face recognition methods may be divided into methods based ontwo-dimensional (2D) photographs and methods based on three-dimensional(3D) modeling. The methods based on 3D modeling are advantageous in thata high recognition rate and stable performance can be achieved invarious environments, but are disadvantageous in that expensiveequipment is required and a large amount of computation for recognitionmakes it difficult to recognize a face in real time. Meanwhile, althoughthe methods based on 2D photographs are disadvantageous in thatrecognition performance is influenced a lot by illumination, thedirection of a face, and the change in facial expression, they areadvantageous in that fast recognition can be accomplished withinexpensive equipment. Accordingly, a method based on 2D photographs ispreferred to a method based on 3D modeling for use in a search system.

Face feature extraction is most essential to a 2D face recognitionsystem based on photographs. Since an original face photograph hashigh-dimensional data, when it is used in the 2D face recognition systemas it is, system efficiency is decreased due to a large amount ofcomputation. For this reason, the 2D face recognition system requiresfeature extraction, in which a face portion suitable for recognition isextracted from an original face image or the original face image isreconstructed into a format suitable for recognition.

For instance, when a black-and-white image having a size of 64×64 pixelsis used in a face recognition system, 64×64=4,096-dimensional data isused. Although a 64×64 pixel image does not seem very big in presentcomputer technology, 4,096-dimensional data is high-dimensional data interms of data analysis and processing with respect to presentcomputation performance. In addition, the original 4,096-dimensionaldata is not necessary to be used as it is since redundant information orunnecessary information (e.g., noise) exists in the pixels.Consequently, to construct an efficient recognition system, dimensionreduction to express 4,096-dimensional image data in a low-dimensionaldata format or feature extraction is needed.

Cases of low-dimensional data or features that can be generated from ahigh-dimensional image or data are almost infinite. Accordingly, apredetermined standard is needed in order to generate and selectsignificant low-dimensional data and optimized low-dimensional data isgenerated based on the predetermined standard. This standard fordimension reduction is referred to as a criterion. Different dimensionreduction methods have different criterions. For example, principalcomponent analysis (PCA), which is the most widely used dimensionreduction method, has a variance of data as the criterion. In otherwords, the higher the variance of data is in a low-dimensional space,the better high-dimensional data is expressed in low dimensions. Asanother example, Fisher linear discriminant (FLD) or Linear discriminantanalysis (LDA) uses as the criterion a ratio between a between-classscatter and a within-class scatter. In other words, high-dimensionaldata is rewritten as low-dimensional data so that the ratio between thetwo scatters is maximum in the low-dimensional.

Actual dimension reduction is accomplished by projectinghigh-dimensional data on a base vector, i.e., a projection vector. Whenhigh-dimensional data is applied to a criterion of the aforementionedcurrent dimension reduction methods, a projection vector that maximizesor minimizes a value of the criterion is calculated. In order to obtainthe projection vector, optimization is used. In many wide-spreaddimension reduction methods, the projection vector can be easilyobtained by solving an eigenvalue problem or a generalized eigenvalueproblem. Accordingly, it can be concluded that a criterion itselfdefines a dimension reduction method.

The biggest problems of 2D-based face recognition, and particularly ofLDA or FLD, are generalization and overfit. Recognition performance issatisfactory with respect to a person whose face image has been usedduring the generation of a base vector but is not satisfactory withrespect to a person whose face image has not been used during thegeneration of the base vector. In addition, when many face images ofdifferent persons are used to generate the base vector, recognitionperformance is higher than in other cases. Also, since a human facechanges little by little over time, when images as recent as possibleare used in base vector generation and feature extraction andregistration, satisfactory recognition performance can be expected.

SUMMARY OF THE INVENTION

The present invention provides a method and apparatus for extracting aface feature, in which a criterion is provided for face featureextraction suitable for face search, thereby effectively obtaining abase vector and a face feature based on the criterion and providing ahigh recognition rate.

The present invention also provides a method and apparatus forextracting a face feature with a small amount of data by generating anew statistic value, a base vector, and a face feature using data of anew face image and an existing statistic value.

According to an aspect of the present invention, there is provided amethod of extracting a face feature. The method includes: calculating anaverage image of each of a plurality of candidates having one or moreface images to which different weights are given according to an orderin which the face images are acquired and calculating a total averageimage of all face images of the plurality of candidates; calculating abetween-class scatter matrix based on a difference between eachcandidate's average image and the total average image and a within-classscatter matrix based on a difference between each candidate's averageimage and each candidate's predetermined face image of the one or moreface images; and generating a base vector that maximizes a criteriondefined by a ratio between the between-class scatter matrix and thewithin-class scatter matrix.

According to another aspect of the present invention, there is providedan apparatus for extracting a face feature. The apparatus includes: anaverage image calculator calculating an average image of each of aplurality of candidates having one or more face images to whichdifferent weights are given according to an order in which the faceimages are acquired and calculating a total average image of all faceimages of the plurality of candidates; a scatter calculator calculatinga between-class scatter matrix based on a difference between eachcandidate's average image and the total average image and a within-classscatter matrix based on a difference between each candidate's averageimage and each candidate's predetermined face image of the one or moreface images; and a base vector generator generating a base vector thatmaximizes a criterion defined by a ratio between the between-classscatter matrix and the within-class scatter matrix.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features and advantages of the present inventionwill become more apparent by describing in detail exemplary embodimentsthereof with reference to the attached drawings in which:

FIG. 1 is a flowchart of a method of extracting a face feature by givinga weight to a face image, according to an embodiment of the presentinvention;

FIG. 2 is a graph for explaining how to calculate an average accordingto an embodiment of the present invention;

FIG. 3 illustrates equations and procedure for updating previousstatistic values and a face feature, according to an embodiment of thepresent invention;

FIG. 4 is a flowchart of a procedure for registering a new candidate andupdating a base vector and pre-registered candidates' face feature(i.e., a template) using image data of the new candidate;

FIG. 5 is a flowchart of a procedure for searching for a candidate toconfirm an identity and updating a base vector and pre-registeredcandidates' face feature using the candidate's image data input duringthe searching;

FIG. 6 is a block diagram of an apparatus for extracting a face featureaccording to an embodiment of the present invention;

FIG. 7 illustrates examples of face images used in a method ofextracting a face feature according to an embodiment of the presentinvention; and

FIG. 8 is a graph showing the comparison between a face recognition rateobtained using conventional linear discriminant analysis (LDA) and aface recognition rate obtained using an embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the attached drawings, in whichlike reference numerals denote like elements. When detailed descriptionof related conventional functions or structures unnecessarily obscurethe gist of the invention, the description will be omitted.

FIG. 1 is a flowchart of a method of extracting a face feature by givinga weight to a face image, according to an embodiment of the presentinvention. Referring to FIG. 1, when a face image is input in operationS110, a weight “α” is given to the face image according to an order inwhich the face image is acquired and the number of images of a candidatecorresponding to the face image and the total number of candidates arecalculated in operation S120. The weight “α” has a value of 0 through 1and has a value close to 1 when the face image is an image acquiredrecently and a value close to 0 when the face image is an image acquiredlong ago. Accordingly, when more new face images are input, a weight toa face image obtained long ago gets closer to 0 and the influence of theface image obtained long ago gets smaller in generation of a basevector. When the face image is of a candidate that has not beenregistered, the total number of candidates increases by 1. When the faceimage is of a pre-registered candidate, the total number of candidatesremains the same.

In operation S130, the candidate's average image and a total averageimage for the total candidates are calculated using the number ofcandidate images and the total number of candidates.

In operation S140, a within-class scatter matrix and a between-classscatter matrix are calculated using each candidate's average image andthe total average image. The within-class scatter matrix is a scattermatrix made using a difference between a piece of image data of acandidate and average image data of the candidate. The between-classscatter matrix is a scatter matrix made using a difference between theaverage image data of the candidate and total average image data. Thebetween-class scatter matrix is calculated without giving a weight basedon the number of face images of each candidate and the within-classscatter matrix is calculated by normalizing the between-class scattermatrix using the number of face images of the candidate so that thescatters are not biased by the number of the candidate's face images,which may be different for each of the individual candidates.

In operation S150, a vector that maximizes a criterion defined by aratio between the within-class scatter matrix and the between-classscatter matrix is generated as a base vector used to extract a facefeature.

In operation S160, each candidate's average face image is projected onthe base vector and a face feature of each candidate is extracted.

When another face image is input, a new base vector is generated usingstatistic values calculated in operations S120 through S150 and data ofthe newly input face image, that is, newly obtained average face imagedata of a current candidate, is projected on the new base vector, and aface feature of the current candidate is updated and extracted. Forother candidates, existing average face image data of each candidate isprojected on the new base vector and a face feature of the candidate isupdated and extracted. When the input face image is not of apre-registered candidate, the input face image is projected on the newbase vector and a face feature is extracted and registered for a newcandidate.

Hereinafter, a method of generating a criterion and a base vector, whichare used to extract a face feature according to an embodiment of thepresent invention, will be described. Principal component analysis andlinear discriminant analysis (LDA) or Fisher linear discriminant (FLD)are representative methods of extracting a face feature. It has beenknown that the LDA, which is a supervised learning method, provides ahigher recognition rate than the principal component analysis, which isan unsupervised learning. The criterion of the LDA is defined byEquation (6) based on a between-class scatter matrix expressed byEquation (1) and a within-class scatter matrix expressed by Equation(4). Each candidate constructs a single class, which includes at leastone face image of the candidate.

$\begin{matrix}{{S_{B} = {\sum\limits_{i = 1}^{c}\;{{n_{i}\left( {m_{i} - m} \right)}\left( {m_{i} - m} \right)^{T}}}},} & (1)\end{matrix}$where “c” is the total number of candidates, n_(i) is the number of faceimages of an i-th candidate, mi is an average face image (hereinafter,an average image) of the i-th candidate, and m is an average of allimages of the total candidates, i.e., a total average face image(hereinafter, a total average image). Here, m_(i) and m are defined byEquations (2) and (3), respectively:

$\begin{matrix}{m_{i} = {\frac{1}{n_{i}}{\sum\limits_{s = 1}^{n_{i}}\;{x_{i,s}.}}}} & (2) \\{m = {{\frac{1}{n}{\sum\limits_{s = 1}^{n}\; x_{s}}} = \left. {\frac{1}{n}{\sum\limits_{i = 1}^{c}\;{n_{i}m_{i}}}}\leftarrow \right.}} & (3)\end{matrix}$where “x” is a vector made by connecting two-dimensional (2D) facephotographs in a row or column, x_(s) is an s-th image in all faceimages, and x_(i,s) is an s-th face image of the i-th candidate.

$\begin{matrix}{S_{W} = \left. {\sum\limits_{i = 1}^{c}S_{i}}\leftarrow \right.} & (4)\end{matrix}$where S_(i) is a within-class scatter of the i-th candidate and isdefined by Equation (5).

$\begin{matrix}{S_{i} = \left. {\sum\limits_{s = 1}^{n_{i}}\;{\left( {x_{i,s} - m_{i}} \right)\left( {x_{i,s} - m_{i}} \right)^{T}}}\leftarrow \right.} & (5) \\{{J(w)} = {\frac{w^{T}S_{B}w}{w^{T}S_{W}w}.}} & (6)\end{matrix}$

Here, “w” is the base vector used to extract a face feature and is avector maximizing a value of Equation (6). As expressed in Equation (7),a face feature “y” is obtained by projecting the face image “x” on thebase vector “w”:y=w^(T)x_(←).   (7)

Here, “w” can be obtained by solving a generalized eigenvalue problemobtained from Equation (6) and a plurality of values of “w” are selectedaccording to an order of the magnitude of an eigenvalue. In other words,“y” also has a vector form. For clarity of the description, a vectorhaving a biggest eigenvalue is represented as the base vector “w”. Thebook “Pattern Classification (2nd Ed),” written by Richard O. Duda,Peter E. Hart, and David G. Stork and published by Wiely-Interscience,can be referred to for details about the LDA.

As shown in Equation (1), the between-class scatter matrix includes adifference between the average image of each class and the total averageimage and indicates how much face images between different persons aredifferent from each other. Accordingly, when the between-class scattermatrix has a bigger value, recognition can be performed more easily.However, the number of face images of the i-th candidate, n_(i), isgiven as a weight to the difference between the average face image ofthe i-th candidate, m_(i), and the total average image “m”. Accordingly,a feature value is biased to a candidate having more face images thanother candidates. Similarly, with respect to the total average image “m”in Equation (3), the number of images of a candidate is given as aweight to an average image of each class.

FIG. 2 is a graph for explaining how to calculate an average accordingto an embodiment of the present invention. FIG. 2 shows a case wherewhen the number of data items in each class is different betweenclasses, and a real average calculated from data is much different froman average cognitively expected, i.e., a cognitive average.

Referring to FIG. 2, an average of a first class is calculated from 100data items and an average of a second class is calculated from 10 dataitems. In the drawing, a total average is calculated using Equation (3)and is close to the average of the first class. In other words, it canbe inferred that the cognitive average is more appropriate than thetotal average in order to extract a feature with differences betweenclasses scattered uniformly. As illustrated in FIG. 2, when newlyacquired images of candidates are used in order to update a basisvector, a base vector is biased toward a candidate having a high usefrequency. Accordingly, a criterion and a method for extracting afeature without being biased by the number of images of each candidateare required.

In the embodiments of the present invention, a between-class scattermatrix defined by Equation (8) obtained by modifying Equation (1) and atotal average defined by Equation (9) obtained by modifying Equation (3)are used in order to extract feature that is not biased by the number ofdata items of each class. The cognitive average shown in FIG. 2 iscalculated using Equation (9).

$\begin{matrix}{S_{B} = {\sum\limits_{i = 1}^{c}\;{\left( {m_{i} - m} \right)\left( {m_{i} - m} \right)^{T}}}} & (8) \\{m = {\frac{1}{c}{\sum\limits_{i = 1}^{c}\; m_{i}}}} & (9)\end{matrix}$

The within-class scatter matrix defined by Equation (4) is calculated bysimply adding each within-class scatter defined by Equation (5) toindicate a cohesion in a class, and therefore, the within-class scattermatrix also may be biased to a class having many data items. To preventsuch biasing, in the embodiments of the present invention, awithin-class scatter of each class is calculated using Equation (10)which is normalized by the number of data items in the class.

$\begin{matrix}{S_{i} = {\sum\limits_{s = 1}^{n_{i}}\;{\frac{1}{n_{i}}\left( {x_{i,s} - m_{i}} \right)\left( {x_{i,s} - m_{i}} \right)^{T}}}} & (10)\end{matrix}$

Since a human face changes little by little over time, a high successratio can be expected when recognition is performed using a face featureextracted from a face image as recent as possible. Accordingly, when abigger weight is given to a more recent image during feature extraction,recognition can be performed more effectively. A k-th candidate'saverage face image with a weight according to order of acquisition timecan be defined by Equation (11).

$\begin{matrix}{m_{k} = {\frac{1}{\lambda_{k}}{\sum\limits_{s = 1}^{n_{i}}\;{\alpha^{n_{i} - s}x_{k,s}}}}} & (11)\end{matrix}$

Here, X_(k,s) is a face an image acquired s-th in time order among faceimages of the k-th candidate. In other words, X_(k,nk) is the k-thcandidate's face image acquired more recently. In Equation (11), α is aconstant having a value of 0 to 1 and λ_(k) is the number of face imagesof the k-th candidate, which is calculated in such a way that a weightis given to each face image according to an acquisition order of theface image. When α is 1, λ_(k) is equal to n_(k) indicating the numberof face images that are really acquired and Equation (11) complies withthe usual definition of an average and λ_(k) is expressed by Equation(12).

$\begin{matrix}{\lambda_{k} = {\sum\limits_{i = 1}^{n_{k}}\;\alpha^{n_{k} - i}}} & (12)\end{matrix}$

Accordingly, when the average face image of the k-th candidate iscalculated using Equation (11), a bigger weight is given to a mostrecent face image. Since α has a value of 0 to 1, when more new faceimages are acquired, a weight of a face image acquired firstly getscloser to 0 and thus the face image acquired firstly hardly has theinfluence on the base vector generation.

The total average face image is calculated using Equation (13) based onEquations (11) and (12).

$\begin{matrix}{m = {\frac{1}{c}{\sum\limits_{i = 1}^{c}\; m_{i}}}} & (13)\end{matrix}$

The between-class scatter matrix is calculated using Equation (14) basedon Equations (11) and (13).

$\begin{matrix}\begin{matrix}{S_{B} = {\sum\limits_{i = 1}^{c}\;{\left( {m_{i} - m} \right)\left( {m_{i} - m} \right)^{T}}}} \\{= {{\sum\limits_{i = 1}^{c}\;{m_{i}m_{i}^{T}}} - {c\; m\; m^{T}}}}\end{matrix} & (14)\end{matrix}$

A within-class scatter of a k-th class is calculated using Equation (15)based on Equation (11).

$\begin{matrix}\begin{matrix}{S_{k} = {\frac{1}{\lambda_{k}}{\sum\limits_{s = 1}^{n_{k}}\;{{\alpha^{n_{i} - s}\left( {x_{k,s} - m_{k}} \right)}\left( {x_{k,s} - m_{k}} \right)^{T}}}}} \\{= {{\frac{1}{\lambda_{k}}{\sum\limits_{s = 1}^{n_{k}}\;{\alpha^{n_{i} - s}x_{k,s}x_{k,s}^{T}}}} - {m_{k}m_{k}^{T}}}}\end{matrix} & (15)\end{matrix}$

The within-class scatter matrix for all face images is calculated usingEquation (16) based on Equation (15).

$\begin{matrix}{S_{W} = {\sum\limits_{i = 1}^{c}\;{S_{i}.}}} & (16)\end{matrix}$

In order to update a base vector and update a feature using the updatedbase vector when a new face image and class information are given,Equations (12) through (16) need to be newly calculated. However,statistic values defined by Equations (12) through (16) are calculatedfrom whole data. In other words, in order to calculate values reflectingnew data, all of old data is also needed. In this case, all face imagesthat has been input must be stored in a system in order to updatestatistic values, which is inefficient in terms of system management. Toovercome this problem, the present invention provide a method ofupdating statistic values using only a new image and previous statisticvalues without using whole data.

The average face image of the k-th candidate can be updated usingEquation (17):

$\begin{matrix}\begin{matrix}{{\overset{\sim}{m}}_{k} = {\frac{1}{{\overset{\sim}{\lambda}}_{k}}{\sum\limits_{s = 1}^{n_{k} + 1}\;{\alpha^{n_{k} + 1 - s}x_{k,s}}}}} \\{= {\frac{1}{{\overset{\sim}{\lambda}}_{k}}\left( {{\sum\limits_{s = 1}^{n_{k}}\;{\alpha^{n_{k} + 1 - s}x_{k,s}}} + x_{k,{n_{k} + 1}}} \right)}} \\{= {\frac{1}{{\overset{\sim}{\lambda}}_{k}}\left( {{\alpha{\sum\limits_{s = 1}^{n_{k}}\;{\alpha^{n_{k} - s}x_{k,s}}}} + x_{k,{n_{k} + 1}}} \right)}} \\{= {{\frac{{\alpha\lambda}_{k}}{{\overset{\sim}{\lambda}}_{k}}\frac{1}{\lambda_{k}}{\sum\limits_{s = 1}^{n_{k}}\;{\alpha^{n_{k} - s}x_{k,s}}}} + {\frac{1}{{\overset{\sim}{\lambda}}_{k}}x_{k,{n_{k} + 1}}}}} \\{= {{\frac{{\alpha\lambda}_{k}}{{\overset{\sim}{\lambda}}_{k}}m_{k}} + {\frac{1}{{\overset{\sim}{\lambda}}_{k}}x_{k,{n_{k} + 1}}}}} \\{= {{\frac{{\overset{\sim}{\lambda}}_{k} - 1}{{\overset{\sim}{\lambda}}_{k}}m_{k}} + {\frac{1}{{\overset{\sim}{\lambda}}_{k}}x_{k,n_{k + 1}}}}} \\{= {m_{k} + {\frac{1}{{\overset{\sim}{\lambda}}_{k}}\left( {x_{k,{n_{k} + 1}} - m_{k}} \right)}}}\end{matrix} & (17)\end{matrix}$where x_(k,nk+1) is a newly acquired n_(k+1)-th image, m_(k) is anaverage image calculated using the previous n_(k) images and is definedby Equation (11), and {tilde over (m)}_(k) is a new average image usingn_(k+1) images including the newly acquired image. As shown in Equation(17), the new average image {tilde over (m)}_(k) can be calculated fromthe previous average image m_(k) and the newly acquired imagex_(k,nk+1). Accordingly, when a new image is acquired, update of theaverage image of the k-th candidate can be efficiently performed withoutstoring all images in the system. Here, {tilde over (λ)}_(k) is definedby Equation (18).{tilde over (λ)}_(k)=αλ_(k)+1   (18)

The total average image is updated using Equations (19) and (20).

Equation (19) is for updating a total average when a new face image of apre-registered candidate is acquired. According to Equation (19), a newtotal average face image is calculated using a newest average image ofthe k-th candidate based on the newly acquired image and a previousaverage image of the k-th candidate.

$\begin{matrix}\begin{matrix}{\overset{\sim}{m} = {\frac{1}{c}{\sum\limits_{i = 1}^{c}\;{\overset{\sim}{m}}_{i}}}} \\{= {{\frac{1}{c}{\sum\limits_{i \neq k}^{c}\; m_{i}}} + {\frac{1}{c}{\overset{\sim}{m}}_{k}}}} \\{= {{\frac{1}{c}{\sum\limits_{i \neq k}^{c}\; m_{i}}} + {\frac{1}{c}m_{k}} - {\frac{1}{c}m_{k}} + {\frac{1}{c}{\overset{\sim}{m}}_{k}}}} \\{= {m + {\frac{1}{c}\left( {{\overset{\sim}{m}}_{k} - m_{k}} \right)}}}\end{matrix} & (19)\end{matrix}$

When a new candidate is registered and a face image of the new candidateis acquired, the total average image is updated using Equation (20):

$\begin{matrix}\begin{matrix}{\overset{\sim}{m} = {\frac{1}{c + 1}{\sum\limits_{i = 1}^{c + 1}\;{\overset{\sim}{m}}_{i}}}} \\{= {{\frac{1}{c + 1}{\sum\limits_{i = 1}^{c}\; m_{i}}} + {\frac{1}{c + 1}{\overset{\sim}{m}}_{c + 1}}}} \\{= {{\frac{c}{c + 1}\frac{1}{c}{\sum\limits_{i = 1}^{c}\; m_{i}}} + {\frac{1}{c + 1}{\overset{\sim}{m}}_{c + 1}}}} \\{= {{\frac{c}{c + 1}m} + {\frac{1}{c + 1}{\overset{\sim}{m}}_{c + 1}}}}\end{matrix} & (20)\end{matrix}$where {tilde over (m)}_(c+1) is an average image of the new candidate.Since it is assumed that only face image of the (c+1)-th candidate isacquired, updated average images of respective first through c-thcandidates are the same as previous average images thereof.

Equation (21) is obtained by combining Equations (19) and (20):

$\begin{matrix}\begin{matrix}{\overset{\sim}{m} = {\frac{1}{\overset{\sim}{c}}\left( {{c\; m} + {\overset{\sim}{m}}_{k} - m_{k}} \right)}} \\{= {\frac{1}{\overset{\sim}{c}}\left\lbrack {{c\; m} + {\frac{1}{{\overset{\sim}{\lambda}}_{k}}\left( {\overset{\sim}{x} - m_{k}} \right)}} \right\rbrack}}\end{matrix} & (21)\end{matrix}$where {tilde over (x)} is a newly acquired face image. Here, {tilde over(c)}=c when the newly acquired face image is of a pre-registeredcandidate, and {tilde over (c)}=c+1 and m_(k)=0 when the newly acquiredface image is of a newly registered candidate.

When a new image of a pre-registered candidate is acquired by thesystem, a new between-class scatter matrix is calculated using Equation(22). When a new image of a newly registered candidate is acquired bythe system, a new between-class scatter matrix is calculated usingEquation (23).

$\begin{matrix}\begin{matrix}{{\overset{\sim}{S}}_{B} = {{\sum\limits_{i = 1}^{c}\;{{\overset{\sim}{m}}_{i}{\overset{\sim}{m}}_{i}^{T}}} - {c\;\overset{\sim}{m}{\overset{\sim}{m}}^{T}}}} \\{= {{\sum\limits_{i \neq k}^{c}\;{m_{i}m_{i}^{T}}} + {{\overset{\sim}{m}}_{k}{\overset{\sim}{m}}_{k}^{T}} - {c\;\overset{\sim}{m}{\overset{\sim}{m}}^{T}}}} \\{= \begin{matrix}{{\sum\limits_{i \neq k}^{c}\;{m_{i}m_{i}^{T}}} + {m_{k}m_{k}^{T}} - {c\; m\; m^{T}} -} \\{{m_{k}m_{k}^{T}} + {c\; m\; m^{T}} + {{\overset{\sim}{m}}_{k}{\overset{\sim}{m}}_{k}^{T}} - {c\;\overset{\sim}{m}{\overset{\sim}{m}}^{T}}}\end{matrix}} \\{= {S_{B} - {m_{k}m_{k}^{T}} + {{\overset{\sim}{m}}_{k}{\overset{\sim}{m}}_{k}^{T}} + {c\; m\; m^{T}} - {c\;\overset{\sim}{m}\;{\overset{\sim}{m}}^{T}}}}\end{matrix} & (22) \\\begin{matrix}{{\overset{\sim}{S}}_{B} = {{\sum\limits_{i = 1}^{c + 1}\;{{\overset{\sim}{m}}_{i}{\overset{\sim}{m}}_{i}^{T}}} - {\left( {c + 1} \right)\overset{\sim}{m}{\overset{\sim}{m}}^{T}}}} \\{= {{\sum\limits_{i = 1}^{c}\;{m_{i}m_{i}^{T}}} + {{\overset{\sim}{m}}_{c + 1}{\overset{\sim}{m}}_{c + 1}^{T}} - {\left( {c + 1} \right)\overset{\sim}{m}{\overset{\sim}{m}}^{T}}}} \\{= \begin{matrix}{{\sum\limits_{i = 1}^{c}\;{m_{i}m_{i}^{T}}} - {c\; m\; m^{T}} + {c\; m\; m^{T}} +} \\{{{\overset{\sim}{m}}_{c + 1}{\overset{\sim}{m}}_{c + 1}^{T}} - {\left( {c + 1} \right)\overset{\sim}{m}{\overset{\sim}{m}}^{T}}}\end{matrix}} \\{= {S_{B} + {{\overset{\sim}{m}}_{c + 1}{\overset{\sim}{m}}_{c + 1}^{T}} + {c\; m\; m^{T}} - {\left( {c + 1} \right)\overset{\sim}{m}{\overset{\sim}{m}}^{T}}}}\end{matrix} & (23)\end{matrix}$

Combination of Equations (22) and (23) produces Equation (24).{tilde over (S)} _(B) =S _(B) −m _(k) m _(k) ^(T) +{tilde over (m)} _(k)^(T) +cm m ^(T) −{tilde over (c)}{tilde over (m)}{tilde over (m)} ^(T)  (24)

As described above, {tilde over (c)}=c when the newly acquired faceimage is of a pre-registered candidate, and {tilde over (c)}=c+1 whenthe newly acquired face image is of a newly registered candidate.

The within-class scatter of the k-th candidate is calculated usingEquation (25).

$\begin{matrix}\begin{matrix}{{\overset{\sim}{S}}_{k} = {{\frac{1}{{\overset{\sim}{\lambda}}_{k}}{\sum\limits_{s = 1}^{n_{k} + 1}\;{\alpha^{n_{k} + 1 - s}x_{k,s}x_{k,s}^{T}}}} - {{\overset{\sim}{m}}_{k}{\overset{\sim}{m}}_{k}^{T}}}} \\{= {{\frac{1}{{\overset{\sim}{\lambda}}_{k}}\left( {{\alpha{\sum\limits_{s = 1}^{n_{k}}\;{\alpha^{n_{k} - s}x_{k,s}x_{k,s}^{T}}}} + {x_{k,{n_{k} + 1}}x_{k,{n_{k} + 1}}^{T}}} \right)} - {{\overset{\sim}{m}}_{k}{\overset{\sim}{m}}_{k}^{T}}}} \\{= \begin{matrix}{{\frac{{\alpha\lambda}_{k}}{{\overset{\sim}{\lambda}}_{k}}\left( {{\frac{1}{\lambda_{k}}{\sum\limits_{s = 1}^{n_{k}}\;{\alpha^{n_{k} - s}x_{k,s}x_{k,s}^{T}}}} - {m_{k}m_{k}^{T}}} \right)} +} \\{{\frac{1}{{\overset{\sim}{\lambda}}_{k}}{\overset{\sim}{x}}_{k,{n_{k} + 1}}x_{k,{n_{k} + 1}}^{T}} + {\frac{{\alpha\lambda}_{k}}{{\overset{\sim}{\lambda}}_{k}}m_{k}m_{k}^{T}} - {{\overset{\sim}{m}}_{k}{\overset{\sim}{m}}_{k}^{T}}}\end{matrix}} \\{= {{\frac{{\overset{\sim}{\lambda}}_{k} - 1}{{\overset{\sim}{\lambda}}_{k}}S_{k}} + {\frac{1}{{\overset{\sim}{\lambda}}_{k}}x_{k,{n_{k} + 1}}x_{k,{n_{k} + 1}}^{T}} + {\frac{{\overset{\sim}{\lambda}}_{k} - 1}{{\overset{\sim}{\lambda}}_{k}}m_{k}m_{k}^{T}} - {{\overset{\sim}{m}}_{k}{\overset{\sim}{m}}_{k}^{T}}}} \\{= {{\frac{{\overset{\sim}{\lambda}}_{k} - 1}{{\overset{\sim}{\lambda}}_{k}}S_{k}} + {\frac{{\overset{\sim}{\lambda}}_{k} - 1}{{\overset{\sim}{\lambda}}_{k}{\overset{\sim}{\lambda}}_{k}}\left( {m_{k} - \overset{\sim}{x}} \right)\left( {m_{k} - \overset{\sim}{x}} \right)^{T}}}} \\{= {\frac{{\overset{\sim}{\lambda}}_{k} - 1}{{\overset{\sim}{\lambda}}_{k}}\left\lbrack {S_{k} + {\frac{1}{{\overset{\sim}{\lambda}}_{k}}\left( {m_{k} - \overset{\sim}{x}} \right)\left( {m_{k} - \overset{\sim}{x}} \right)^{T}}} \right\rbrack}}\end{matrix} & (25)\end{matrix}$

When a new image of a pre-registered candidate is acquired by thesystem, a new within-class scatter matrix is calculated using Equation(26). When a new image of a newly registered candidate is acquired bythe system, a new within-class scatter matrix is calculated usingEquation (27).

$\begin{matrix}\begin{matrix}{{\overset{\sim}{S}}_{W} = {\sum\limits_{i = 1}^{c}\;{\overset{\sim}{S}}_{i}}} \\{= {{\sum\limits_{i \neq k}^{c}\; S_{i}} + {\overset{\sim}{S}}_{k}}} \\{= {{\sum\limits_{i \neq k}^{c}\; S_{i}} + S_{k} - S_{k} + {\overset{\sim}{S}}_{k}}} \\{= {{\sum\limits_{i = 1}^{c}\; S_{i}} - S_{k} + {\overset{\sim}{S}}_{k}}} \\{= {S_{W} - S_{k} + {\overset{\sim}{S}}_{k}}}\end{matrix} & (26) \\\begin{matrix}{{\overset{\sim}{S}}_{W} = {\sum\limits_{i = 1}^{c + 1}\;{\overset{\sim}{S}}_{i}}} \\{= {{\sum\limits_{i = 1}^{c}\; S_{i}} + {\overset{\sim}{S}}_{c + 1}}} \\{= {S_{W} + {\overset{\sim}{S}}_{c + 1}}}\end{matrix} & (27)\end{matrix}$

Equation (28) is obtained by combining Equations (26) and (27). Whenupdate is performed based on the new image of the newly registeredcandidate, S_(k)=0.{tilde over (S)} _(W) =S _(W) −S _(k) +{tilde over (S)} _(k)   (28)

FIG. 3 illustrate equations and procedure for updating previousstatistic values and a face feature calculated by the method illustratedin FIG. 1, according to an embodiment of the present invention.Referring to FIG. 3, {tilde over (x)} is a newly acquired face image instage 310. For clarity of the description, it is assumed that {tildeover (x)} is an image of a k-th candidate. Next, the previous number offace images of the k-th candidate, which is calculated in such a waythat a weight is given according to the acquisition order, is updated bygiving a new weight in stage 320. The total number of candidates isupdated in stage 325. If {tilde over (x)} is an image of apre-registered candidate, the total number of candidates does notchange. However, when {tilde over (x)} is the image of a newlyregistered candidate, the total number of candidates increases by 1.Next, the average image of the k-th candidate is updated in stage 330and the total average image with respect to all candidates is alsoupdated in stage 340. The within-class scatter of the k-th candidate,the between-class scatter matrix, and the within-class scatter matrixfor all candidates are updated in stages 350, 360, and 370,respectively, using the statistic values obtained in the previousstages. The base vector “w” used to extract a face feature is calculatedfrom the updated between-class scatter matrix and the updatedwithin-class scatter matrix in stage 380. A new face feature reflectingthe newly acquired face image is extracted for each of all candidatesusing the base vector “w” and the average image of each candidate instage 390. Accordingly, a face feature of a candidate corresponding tothe newly acquired face image is updated using a new average image ofthe candidate and a new base vector while a face feature of each ofother candidates is updated using an existing old average image of eachcandidate and the new base vector.

FIG. 4 is a flowchart of a procedure for registering a new candidate andupdating a base vector and a pre-registered candidate's face feature(i.e., template) using information on the new candidate, which is inputduring the registration. In operation S410, registration of a newcandidate is requested. In operation S420, a face image of the newcandidate is acquired using, for example, a camera. In operation S430,personal information on the new candidate other than the face image isregistered. In operation S440, an operator finally confirms theregistration.

In operation S450, information used to update a face feature of each ofall registered candidates is generated using the input face image of thenew candidate and the finally confirmed information on the newcandidate. Here, the most important information is the face image of thenew candidate and class information of the face image. An average faceimage of the new candidate and a total average face image are calculatedbased on the face image of the new candidate and are used to calculate awithin-class scatter matrix and a between-class scatter matrix.

In operation S460, a new base vector is generated using the calculatedscatter matrixes. In operation S470, using the new base vector, a facefeature of the new candidate is generated and a face feature of eachpre-registered candidate is updated.

FIG. 5 is a flowchart of a procedure for searching for a candidate andupdating a base vector and a pre-registered candidate's face featureusing the candidate information input during the searching.

In operation S510, confirming of the identity of a candidate isrequested. In operation S520, a face image of the candidate is acquired.In operation S530, a face feature of the candidate is extracted using anexisting base vector. In operation S540, the candidate is searched forusing a 1:N comparison, in which the face feature of the candidate to besearched for is compared with face features of N candidates registeredin a database, and a candidate list for the searched candidate is outputas a result of the searching. In operation S550, an operator comparesthe identity of the candidate with candidate information on thecandidate list and confirms the identity of the candidate. Generally,during the searching, the candidate list including a plurality ofcandidates listed in order of decreasing probability is generated andconfirming of the identity is completed by the operator selecting acandidate from the candidate list, who is decided by the operator to bemost similar to the candidate whose identity is requested to beconfirmed.

In operation S560, update information is generated using the face imageof the candidate, which is acquired during the searching, and identityinformation confirmed by the operator. Here, the most importantinformation is the acquired face image of the candidate and classinformation of the face image. The face image of the candidate whoseidentity has been confirmed is added to a corresponding class, and thenan average face image of the candidate and a total average face imageare calculated. A between-class scatter matrix is calculated based on adifference between the candidate's average face image and the totalaverage face image. A within-class scatter matrix is calculated based ona difference between a face image of the candidate and the candidate'saverage face image.

In operation S570, a new base vector is generated using theabove-calculated within-class scatter matrix and between-class scattermatrix. In operation S580, a face feature of the registered candidatesis updated using the new base vector.

FIG. 6 is a block diagram of an apparatus for extracting a face featureaccording to an embodiment of the present invention. The apparatus willbe described in conjunction with FIGS. 1 to 6. The apparatus includes anaverage image calculator 610, a scatter calculator 620, a base vectorgenerator 630, a face feature extractor 640, a face feature database(DB) 645, and a comparison search unit 650.

The average image calculator 610 calculates a candidate's average imageand a total average image with respect to input face images. Since ahuman face changes little by little over time, when a face featureextracted from an image as recent as possible is used, a recognitionrate is increased. Accordingly, a bigger weight is given to a morerecent image when the candidate's average image and the total averageimage are calculated. The calculation is performed using Equations (11)through (13), where the weight “α” has a value of 0 to 1.

When a new face image is input later, the candidate's average image andthe total average image are updated using previous statistic value anddata of the new face image. Calculation for the update is performedusing Equations (17) through (21).

The scatter calculator 620 calculates a between-class scatter matrix anda within-class scatter matrix based on each candidate's average imageand the total average image using Equations (14) through (16). Since thenumber of images of each candidate may be different and a scatter may bebiased to a candidate having a large number of images, a weightaccording to the number of each candidate's images is not given when abetween-class scatter is calculated and while normalization using thenumber of each candidate's images is performed when a within-classscatter is calculated.

When a new face image is input later, the between-class scatter matrixand the within-class scatter matrix are updated using a previousstatistic value and data of the new face image. Calculation for theupdate is performed using Equations (22) through (28).

The base vector generator 630 generates a base vector that maximizes acriterion defined by a ratio between the between-class scatter matrixand the within-class scatter matrix. A plurality of base vectors may begenerated. When a new face image is input later, a new base vector isgenerated using updated statistic values.

The face feature extractor 640 extracts a face feature of each candidateby projecting the candidate's average image on the new base vector. Whenthe input face image is of a newly registered candidate, a face featureof the candidate is extracted by projecting data of the new face imageon the new base vector and a face feature of each pre-registeredcandidate is updated by projecting the existing candidate's averageimage on the new base vector.

Extracted face features and updated face features are stored in the facefeature DB 645.

The comparison search unit 650 compares a face feature extracted byprojecting a face image, which is input for confirmation of the identityof a pre-registered candidate, on an existing base vector with a facefeature of each candidate stored in the face feature DB 645 and outputsa candidate list as a result of the comparison.

FIG. 7 illustrates examples of face images used to test the recognitionrate of a method according to an embodiment of the present invention. Inthe test, 20 images for each of 55 persons (i.e., candidates), that is,1100 images were used. The size of each image was 64×64 pixels. 5 imagesper person were used to calculate an average image of each person and aface feature of the person was extracted and registered based on theaverage image. The remaining 15 images per person were used to test therecognition rate.

FIG. 8 is a graph showing the comparison between a face recognition rateobtained using the conventional LDA and a face recognition rate obtainedusing an embodiment of the present invention. The graph shows the changein the recognition rate with respect to the number of features. In bothof the conventional LDA and the method according to the embodiment ofthe present invention, principal component analysis is used aspre-processing for efficient feature extraction, which is not specifiedin the present invention.

Referring to FIG. 8, line A indicates the recognition rate obtainedusing the conventional LDA while line B indicates the recognition rateobtained when a candidate's face feature is updated every time when animage of the candidate is acquired by using the method of the presentinvention.

Every time when a test is finished for a candidate, a newly acquiredimage is reflected to update a base vector and a face feature, which areused in a subsequent test and updated again. In the embodiment of thepresent invention, a most recent image of a candidate is continuouslyreflected, so that the recognition rate is effectively increased.

The invention can also be embodied as computer readable codes on acomputer readable recording medium. The computer readable recordingmedium is any data storage device that can store data which can bethereafter read by a computer system. Examples of the computer readablerecording medium include read-only memory (ROM), random-access memory(RAM), CD-ROMs, magnetic tapes, floppy disks, optical data storagedevices, and carrier waves (such as data transmission through theInternet). The computer readable recording medium can also bedistributed over network coupled computer systems so that the computerreadable code is stored and executed in a distributed fashion. Also,functional programs, codes, and code segments for accomplishing thepresent invention can be easily construed by programmers skilled in theart to which the present invention pertains.

According to the present invention, a face feature is extracted andupdated in such a way that data of a most recent face image is reflectedand a weight is given according to an order in which a face image isacquired, so that a face recognition rate is increased. In addition, inorder to prevent that the face feature is biased to a certain candidatebecause the numbers of face images are different between candidatesaccording to how often a candidate uses a face recognition system, anequation for calculating a within-class scatter and an equation forcalculating a between-class scatter are adjusted and a base vector isgenerated when the face feature is extracted. Accordingly, accurate facerecognition can be accomplished. Furthermore, since new statistic valuesare obtained using existing statistic values and data of a new inputimage, it is not necessary to store data of all face images that hasbeen input, thereby reducing system cost and facilitating systemmanagement.

While this invention has been particularly shown and described withreference to preferred embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the spirit and scope of theinvention as defined by the appended claims. The preferred embodimentsshould be considered in descriptive sense only and not for purposes oflimitation. Therefore, the scope of the invention is defined not by thedetailed description of the invention but by the appended claims, andall differences within the scope will be construed as being included inthe present invention.

1. A method of extracting a face feature, the method comprising:calculating an average image of each of a plurality of candidates havingtwo or more face images and calculating a total average image of allface images of the plurality of candidates, wherein a recent face imagehas a higher weight value than a least recent face image, wherein eachof the two or more face images has weight value based on the order inwhich each the face images are acquired, wherein a most recent faceimage has a highest weight value and a most least recent face image hasa lowest weight value, wherein the face images between the most recentface image and the most least face image are proportionally weightedbetween the highest weight value the lowest weight value according to anorder, wherein the order is based on when the two or more face imageswere acquired, wherein the weight values for each of the two or moreface images decease proportionally according to a new order when a newmost recent face image is acquired; calculating a between-class scattermatrix based on a difference between each candidate's average image andthe total average image and a within-class scatter matrix based on adifference between each candidate's average image and each candidate'spredetermined face image of the one or more face images; and generatinga base vector that maximizes a criterion defined by a ratio between thebetween-class scatter matrix and the within-class scatter matrix,wherein the face image have a higher weight value has a greaterinfluence on the generation of the base vector, wherein each of the faceimages have a less influence on the base vector changes each time a nextnew most recent face image is acquired.
 2. The method of claim 1,further comprising extracting a face feature by projecting eachcandidate's average image on the base vector.
 3. The method of claim 2,further comprising, when a face image of a new candidate other than theexisting candidates is input, extracting a face feature used to registerthe new candidate by projecting the input face image on the base vector.4. The method of claim 2, further comprising, when a face image of onecandidate among the existing candidates is input: extracting a facefeature used to confirm the identity of the candidate by projecting theinput face image on the base vector; comparing the extracted facefeature with face features of the existing candidates; and outputting acandidate list according to a result of the comparison.
 5. The method ofclaim 3 or 4, further comprising updating previous statistic valueswhich are values calculated in each step based on the previous statisticvalues and the input face image.
 6. The method of claim 1, wherein theweight has a value of 0 to 1 and has a value close to 1 when the imageis more recent.
 7. The method of claim 1, wherein the calculating of thebetween-class scatter matrix and the within-class scatter matrixcomprises calculating a between-class scatter without giving a weightaccording to the number of each candidate's face images to prevent thescatter from being biased by the number of the candidate's face imagesand calculating a within-class scatter using normalization using thenumber of the candidate's face images.
 8. An apparatus for extracting aface feature, the apparatus comprising: an average image calculatorcalculating an average image of each of a plurality of candidates havingtwo or more face images and calculating a total average image of allface images of the plurality of candidates, wherein a recent face imagehas a higher weight value than a least recent face image, wherein eachof the two or more face images has weight value based on the order inwhich each the face images are acquired, wherein a most recent faceimage has a highest weight value and a most least recent face image hasa lowest weight value, wherein the face images between the most recentface image and the most least face image are proportionally weightedbetween the highest weight value the lowest weight value according to anorder, wherein the order is based on when the two or more face imageswere acquired, wherein the weight values for each of the two or moreface images decease proportionally according to a new order when a newmost recent face image is acquired; a scatter calculator calculating abetween-class scatter matrix based on a difference between eachcandidate's average image and the total average image and a within-classscatter matrix based on a difference between each candidate's averageimage and each candidate's predetermined face image of the one or moreface images; and a base vector generator generating a base vector thatmaximizes a criterion defined by a ratio between the between-classscatter matrix and the within-class scatter matrix, wherein the faceimage have a higher weight value has a greater influence on thegeneration of the base vector, wherein each of the face images have aless influence on the base vector changes each time a next new mostrecent face image is acquired.
 9. The apparatus of claim 8, furthercomprising a face feature extractor extracting a face feature byprojecting each candidate's average image on the base vector.
 10. Theapparatus of claim 9, wherein, when a face image of a new candidateother than the existing candidates is input, the face feature extractorextracts a face feature used to register the new candidate by projectingthe input face image on the base vector.
 11. The apparatus of claim 8,wherein the weight has a value of 0 to 1 and has a value close to 1 whenthe image is more recent.
 12. The apparatus of claim 8, wherein thescatter calculator calculates a between-class scatter without giving aweight according to the number of each candidate's face images toprevent the scatter from being biased by the number of the candidate'sface images and calculates a within-class scatter using normalizationusing the number of the candidate's face images.
 13. The apparatus ofclaim 9, further comprising a comparison search unit comparing a facefeature used to confirm the identity of one candidate among existingcandidates with face features of the existing candidates when a faceimage of the candidate is input and outputting a candidate listaccording to a result of the comparison, wherein the face featureextractor extracts the face feature used to confirm the identity of thecandidate by projecting the input face image on the base vector.
 14. Theapparatus of claim 10 or 13, wherein previous statistic values which arevalues calculated by each component of the apparatus are updated basedon the previous statistic values and the input face image.